TruCluster Production Server Software allows you to configure one or more available server environments (ASEs), depending on the number of systems in the cluster. TruCluster Available Server Software allows you to configure one ASE. An ASE uses shared SCSI buses, external storage shelves or RAID controllers, and supports disk mirroring, and fast file system recovery to provide high data availability and reliability.
This chapter introduces SCSI bus configuration concepts and describes requirements for the shared SCSI bus. In addition to using only the supported hardware described in the TruCluster Software Products Release Notes, adhering to the requirements described in this chapter will ensure that your ASE operates correctly.
This chapter discusses the following topics:
Shared SCSI bus configuration requirements (Section 3.1)
Numbering SCSI devices (Section 3.2)
SCSI bus performance (Section 3.3)
SCSI bus device identification numbers (Section 3.4)
SCSI bus length (Section 3.5)
Using SCSI signal converters (Section 3.6)
SCSI bus termination (Section 3.7)
Connecting devices to a shared SCSI bus (Section 3.8)
In some sections of this manual, a distinction is made between the different implementations of the Small Computer System Interconnect (SCSI), SCSI, Fast 10 (F10), and UltraSCSI. Most of the manual uses SCSI as the generic term, which could be any of the SCSI implementations.
A shared SCSI bus must adhere to the following requirements:
Only an external bus can be used for a shared SCSI bus.
The logical bus numbers you assign to the adapters for shared buses must be higher than the bus numbers for local buses. The starting number for shared SCSI buses must be high enough to allow for expansion of local buses without requiring the shared buses to be renumbered.
SCSI bus specifications limit the number of devices on an 8-bit (narrow) SCSI bus to 8. The limit is 16 devices on a 16-bit SCSI bus (wide). However, although Available Server and Production Server supports wide, differential and UltraSCSI devices, DIGITAL UNIX Version 4.0 supports only 8 devices on a SCSI bus. See Section 3.4 for more information.
The length of each physical bus is strictly limited. See Section 3.5 for more information.
For each physical bus or UltraSCSI bus segment, you can have only two terminators, one at each end.
Use trilink connectors and Y cables to connect devices to a shared bus, so you can disconnect the devices without affecting bus termination. See Section 3.7 for more information.
You can directly connect devices only if they have the same transmission mode (differential or single-ended) and data path (narrow or wide). Use a SCSI signal converter to connect devices with different transmission modes or data paths. See Section 3.6 for information about SCSI signal converters.
You can use up to 30 shared buses in an ASE, and a shared SCSI bus can be in only one ASE.
A member system can be in only one ASE, but you can set up multiple ASEs in a TruCluster Production Server Software environment, if you have four or more member systems.
Be careful when performing maintenance on any device that is on a shared bus because of the constant activity on the bus. Usually, to perform maintenance on a device without shutting down the ASE, you must be able to isolate the device from the shared bus without affecting bus termination.
All DIGITAL UltraSCSI host adapters support UltraSCSI disks at UltraSCSI speeds in UltraSCSI BA356 shelves or behind the HSZ70 RAID controller. Older, non-ULtraSCSI BA356 shelves are supported with UltraSCSI host adapters and host RAID controllers as long as they contain no UltraSCSI disks.
UltraSCSI drives and fast wide drives can be mixed together in an UltraSCSI BA356 shelf.
Differential UltraSCSI adapters may be connected to both a non-UltraSCSI BA356 shelf (via a DWZZB-VW) and the new UltraSCSI BA356 shelf (via the DS-BA35X-DA personality module) on the same shared SCSI bus. The UltraSCSI adapter negotiates maximum transfer speeds with each SCSI device.
The HSZ70 UltraSCSI RAID controller has a wide differential UltraSCSI host bus with a Very High Density Cable Interconnect (VHDCI) connector. HSZ70 controllers will work with fast, wide differential SCSI adapters (for example, KZPSA) at fast SCSI speeds. A converter cable is necessary.
Fast, wide SCSI drives (green StorageWorks building blocks (SBBs) with part numbers ending in -VW) may be used in an UltraSCSI BA356 shelf.
Do not use fast, narrow SCSI drives (green SBBs with part numbers ending in -VA) in any shelf that could assign the drive a SCSI ID greater than 7. It will not work.
The UltraSCSI BA356 requires a 180-watt power supply (BA35X-HH). It will not function properly with the older, lower wattage BA35X-HF universal 150-watt power supply.
An older BA356 that has been retrofitted with a BA35X-HH 180-watt power supply and BA35X-DA personality module is not the same as an UltraSCSI BA356.
All member systems must recognize the disks on a shared bus at the same device number. A device number is obtained from the logical bus number, which is defined in the system configuration file. If you connect a shared bus to SCSI controllers that have the same logical bus number on each system, the shared disks will have the same device number on each system.
Bus numbers are assigned to SCSI controllers during the kernel configuration
process, and they are specified in the kernel configuration file. When you
configure a kernel by running the doconfig
program using the
generic kernel, an algorithm is used to probe the SCSI controllers installed in
the system. As the probe algorithm encounters the adapters, it assigns logical
bus numbers to the SCSI controllers in sequence, starting with 0.
Because systems have different numbers of internal buses, in an available
server environment (ASE), the ase_fix_config
script is used to
manually assign logical bus numbers and override the probe algorithm. If you
choose to have the software installation procedure rebuild your kernel, it
automatically runs the script and prompts you for information about bus numbers.
You can also manually invoke the script with the following command:
# /var/ase/sbin/ase_fix_config -c CONFIG-FILE
CONFIG-FILE is the name of the system's configuration file, which
is stored in the /sys/conf
directory.
Before you install your SCSI controllers, plan your bus configuration. DIGITAL recommends that you install SCSI controllers for the local buses in the lower bus slots in your system, leave some empty slots, and then install the SCSI controllers for the shared buses. If you use this method, you can install additional SCSI controllers for both the local buses and the shared buses without disrupting the shared bus numbering scheme. For example, if a maximum of eight local controllers will be installed in a system, install the first controller for a shared bus in slot 8. Then, install another controller for a shared bus in the next highest slot (7), and so on.
See the TruCluster Software Products Software Installation
manual for more information about the ase_fix_config
script. In addition, some
SCSI controllers have two ports (or channels) so you can connect two shared SCSI
buses to each controller. For example, a PMAZC TURBOchannel SCSI controller has
ports A and B, and a KZMSA XMI-to-SCSI adapter has channels 0 and 1. Both ports
do not have to be used, but any unused ports must be terminated.
If you are using dual-port SCSI controllers, a shared bus must be connected to the same port on each system. For example, a shared bus that is connected to port A (or channel 0) on one SCSI controller must be attached to port A (or channel 0) on all the other SCSI controllers.
Before you set up a SCSI bus, it is important that you understand a number of issues that affect the viability of a bus and how the devices connected to it operate. Specifically, bus performance is influenced by the following factors:
Transmission method
Data path
Bus speed
The following sections describe these factors.
Two transmission methods can be used in a SCSI bus:
Single-ended--In a single-ended SCSI bus, one data lead and one ground lead are utilized for the data transmission. A single-ended receiver looks only at the signal wire as the input. The transmitted signal arrives at the receiving end of the bus on the signal wire somewhat distorted by signal reflections. The length and loading of the bus determine the magnitude of this distortion. This transmission method is economical, but is more susceptible to noise than the differential transmission method, and requires short cables. Devices with single-ended SCSI devices include the following:
PMAZC (Available Server only)
KZMSA (Available Server only)
BA350, BA353, and BA356 storage shelves
Single-ended side of a SCSI signal converter or personality module
Differential--Differential signal transmission uses two wires to transmit a signal. The two wires are driven by a differential driver that places a signal on one wire (+SIGNAL) and another signal that is 180 degrees out of phase (-SIGNAL) on the other wire. The differential receiver generates a signal output only when the two inputs are different. As signal reflections occur virtually the same on both wires, they are not seen by the receiver, because it only sees differences on the two wires.
This transmission method is less susceptible to noise than single-ended SCSI, enables you to use longer cables, and uses 68-pin, high-density or 68-pin VHDCI connectors (UltraSCSI). Devices with differential SCSI interfaces include the following:
KZPSA
KZTSA (Available Server only)
HSZ10 (Available Server only), HSZ40, HSZ50, and HSZ70 controllers
Differential side of a SCSI signal converter or personality module
You cannot use the two transmission methods in the same physical bus. For example, a device with a differential SCSI interface must be connected to another device with a differential SCSI interface. If you want to connect devices that use different transmission methods, use a SCSI signal converter between the devices. See Section 3.6 for information about using SCSI signal converters.
UltraSCSI does not support the use of a signal converter such as the DWZZA or DWZZB for Available Server or Production Server Version 1.5. The DS-BA35X-DA personality module functions as a signal converter for the UltraSCSI BA356. It is the interface between the shared differential UltraSCSI bus and the BA356 internal single-ended SCSI bus.
There are two possible data paths for SCSI devices:
Narrow--Implies an 8-bit data path for SCSI-2. The performance of this mode is limited.
Wide--Implies a 16-bit data path for SCSI-2 and a 32-bit data path for UltraSCSI. This mode increases the amount of data that is transferred in parallel on the bus.
Usually, but not always, single-ended devices are narrow, and differential devices are wide. The BA356 storage shelves are examples of single-ended, wide devices. If you want to connect devices that have different data paths, use a SCSI signal converter between the devices.
Bus speeds vary depending upon the type of SCSI bus: SCSI, SCSI-2, fast, wide SCSI-2, or UltraSCSI and the data path width. The following sections discuss these bus speeds in more detail.
SCSI and SCSI-2 controllers operate in two data transfer speeds: standard or slow mode and fast bus speed. In slow mode, the narrow 8-bit SCSI bus achieves up to 5 MB/sec. Fast SCSI-2 bus speed (the fast synchronous transfer option) enables I/O devices to attain high peak-rate transfers in synchronous mode. Production Server supports only fast bus mode.
To set the bus speed on a SCSI controller, use either console commands or the Loadable Firmware Update (LFU) utility, depending on the type of SCSI controller.
Although fast SCSI bus speed (sometimes referred to as Fast 10) doubles the transmission rate to 10 MB/sec, it reduces the maximum cable length for each single-ended bus from 6 meters to 3 meters. Fast bus speed on a wide (16-bit) SCSI bus increases the maximum bus bandwidth to 20 MB/sec (referred to as Fast 20). See Section 3.5 for more information about bus length.
Fast bus speed in an UltraSCSI bus increases the maximim bus bandwidth to 40 MB/sec.
On a shared SCSI bus, each SCSI device uses a device address and must have a unique SCSI ID (from 0 to 7). For example, each SCSI bus adapter and each disk in a single-ended storage shelf uses a device address.
SCSI bus adapters have a default SCSI ID that you can change by using console commands or utilities. For example, a KZPSA adapter has an initial SCSI ID of 7.
Use the following priority order to assign SCSI IDs to the SCSI bus adapters connected to a shared SCSI bus:
7-6-5-4-3-2-1-0
This order specifies that 7 is the highest priority, and 0 is the lowest priority. When assigning SCSI IDs, use the highest priority ID for member systems (starting at 7). Use lower priority IDs for disks.
The SCSI ID for a disk in a BA350 or BA356 storage shelf corresponds to its slot location. In addition, you can set the SCSI IDs for some types of disks in a BA350 or BA356 storage shelf by using switches in the rear of some StorageWorks building blocks (SBBs). The SCSI ID for a disk in a BA353 storage shelf is set by the device address switches on the rear of the box.
The HSZ10 controller uses only one SCSI ID. You set the SCSI ID for an HSZ10 controller using switches on the unit. You can configure the HSZ40 controller with one to four SCSI target addresses.
There is a limit to the length of the cables in a shared SCSI bus. The total cable length for a physical bus or UltraSCSI bus segment is calculated from one terminated end to the other.
If you are using devices that have the same transmission method and data path (for example, wide differential), a shared bus will consist of only one physical bus. If you have devices with different transmission methods, you will have both single-ended and differential physical buses, each of which must be terminated only at both ends and must adhere to the rules on bus length.
Note
In an Available Server or Production Server configuration you always have single-ended SCSI bus segments since all of the storage shelves use a single-ended bus.
Table 3-1 describes the maximum cable length for a physical SCSI bus.
SCSI Bus | Bus Speed | Maximum Cable Length |
Narrow, single-ended | 5 MB/second | 6 meters |
Narrow, single-ended fast | 10 MB/second | 3 meters |
Wide differential, fast | 20 MB/second | 25 meters |
Differential UltraSCSI | 40 MB/second | 24 meters [Footnote 6] |
The total length of a physical bus must include the amount of cable that is located inside each system and disk storage shelf. This length varies, depending on the device. For example, the length of cable inside a BA350, BA353, or BA356 storage shelf is approximately 1.0 meter.
Because of the cable length limit, you must plan your hardware configuration carefully, and ensure that each SCSI bus meets the cable limit guidelines. In general, you must place systems and storage shelves as close together as possible and choose the shortest possible cables for the shared bus.
A SCSI signal converter allows you to couple a differential bus segment to a single-ended bus segment, allowing the mixing of differential and single-ended devices on the same bus to isolate bus segments for maintenance purposes.
Each SCSI signal converter has a single-ended side with either a narrow or a wide data path, and a differential side with a wide data path.
Note
Some UltraSCSI documentation uses the UltraSCSI "bus expander" term when referring to the DWZZA, DWZZB, and UltraSCSI signal converters. Other UltraSCSI documentation refers to some UltraSCSI products as bus extender/converters.
For Available Server and Production Server Version 1.5 there are no supported UltraSCSI bus expanders (DWZZC).
In this manual, any device that converts a differential signal to a single-ended signal is referred to as a signal converter, except the DS-BA35X-DA personality module. However, because the DS-BA35X-DA personality module does convert a differential bus to a single-ended bus, it is discussed with the signal converters.
A SCSI signal converter is required in the following cases:
You want to connect devices with different transmission modes or data path widths.
You are using a DEC 7000 or DEC 10000 system in an available server environment (ASE). The KZMSA SCSI adapter does not have removable termination resistors.
You want to run a PMAZC TURBOchannel SCSI controller using fast SCSI bus speed.
Signal converters can be standalone units, StorageWorks building blocks (SBBs) that are installed in a storage shelf disk slot, or as with the UltraSCSI BA356, the SBB Shelf Differential I/O Module (pesonality module DS-BA35X-DA). You must use the signal converter or personality module that is appropriate for your hardware configuration.
For example, use a DWZZA-VA signal converter to connect a BA350 or BA353 (single-ended and narrow) storage shelf to a differential device, but use a DWZZB-VW signal converter to connect a non-UltraSCSI BA356 (single-ended and wide) storage shelf to a differential adapter. The DS-BA35X-DA personality module is used in an UltraSCSI BA356 to connect a differential adapter to the single-ended disks in the BA356.
Table 2-7 shows the types of supported SCSI signal converters.
The DWZZA and DWZZB signal converters have removable termination. The DS-BA35X-DA personality module has switch selectable termination. The following sections describe these signal converters in more detail.
Both the single-ended side and the differential side of each DWZZA and DWZZB signal converter has removable termination. To use a signal converter, you must remove the termination in the differential side and attach a trilink connector to this side. To remove the differential termination, remove the five 14-pin SIP resistors (located near the differential end of the signal converter). You can terminate the trilink connector to terminate the differential bus. If you detach the trilink connector from the signal converter, the shared SCSI bus is still terminated.
In most cases, you must keep the termination in the single-ended side to terminate the single-ended bus connecting the signal converter and the single-ended device. Verify that the termination is active. A DWZZA should have jumper J2 installed. Jumpers W1 and W2 should be installed in a DWZZB.
However, if you have a DWZZA-VA signal converter installed in a BA353 storage shelf, you must remove the termination from the single-ended side. To do this, remove jumper J2.
Figure 3-1 shows the status of internal termination for a standalone SCSI signal converter that has a trilink connector attached to the differential side.
Figure 3-2 shows the status of internal termination for an SBB SCSI signal converter that has a trilink connector attached to the differential side.
The UltraSCSI BA356 shelf uses a 16-bit differential UltraSCSI personality module (DS-BA35X-DA) as the interface between the UltraSCSI differential bus and the UltraSCSI single-ended bus in the BA356.
The personality module controls termination for the external differential UltraSCSI bus segment, and for both ends of the internal UltraSCSI single-ended bus segment.
For normal cluster operation, the differential termination must be disabled since a trilink connector will be installed on personality module connector JA1, which allows the use of the BA356 in the middle of the bus or external termination for a BA356 on the end of the bus.
Switch pack 4 switches S4-1 and S4-2 are set to ON to disable the personality module differential termination. The switches have no effect on the BA356 internal, single-ended UltraSCSI bus termination.
Note
S4-3 and S4-4 have no function on the DS-BA35X-DA personality module.
Figure 3-3 shows the relative positions of the two DS-BA35X-DA switch packs.
You must properly connect devices to a shared SCSI bus. In addition, you can terminate only the beginning and end of each bus segment (either single-ended or differential).
There are two rules for SCSI bus termination:
Threr are only two terminators for each bus segment.
If it is possible, bus termination must be external.
Note
Generally, when using tape loaders on a shared SCSI bus, cluster SCSI bus termination principles will not be adhered to. The SCSI bus is not terminated externally to the tape loader. Therefore, the tape loader must be on the end of the shared SCSI bus.
Whenever possible, connect devices to a shared bus so that they can be isolated from the bus. This allows you to disconnect devices from the bus for maintenance purposes without affecting bus termination and cluster operation. You also can set up a shared SCSI bus so that you can connect additional devices at a later time without affecting bus termination.
Most devices have internal termination or some other method of termination. For example, PMAZC, KZTSA, and KZPSA adapters, BA350 and BA356 storage shelves, SCSI signal converters, and the UltraSCSI BA356 personality module (DS-BA35X-DA) have internal termination, but BA353 and HSZ40 controllers have automatic termination. Depending on how you set up a shared bus, you may have to enable or disable device termination.
If you use a device's internal termination to terminate a shared bus, and you disconnect the bus cable from the device, the bus will not be terminated and cluster operation will be impaired. Therefore, you must use external termination, enabling you to detach the device without affecting the bus termination.
To be able to externally terminate a bus and connect and disconnect devices without affecting bus termination, remove the device termination and use Y cables or trilink connectors to connect a device to a shared SCSI bus.
By attaching a Y cable or trilink connector to an unterminated device, you can locate the device in the middle or at the end of the shared bus. If the device is at the end of a bus, attach a terminator to the Y cable or trilink connector to terminate the bus. If you disconnect the Y cable or trilink connector from the device, the shared bus is still terminated and the shared SCSI bus is still operable.
In addition, you can attach a Y cable or a trilink connector to a shared bus without connecting the Y cable or trilink connector to a device. If you do this, you can connect a device to the Y cable or trilink connector at a later time without affecting bus termination. This allows you to expand your configuration without shutting down the cluster.
Figure 3-4 shows a BN21V-0B Y cable, which you attach to a PMAZC TURBOchannel SCSI controller that has had its onboard termination removed.
Figure 3-5 shows a BN21W-0B Y cable, which you may attach to a KZTSA or KZPSA SCSI adapter that has had its onboard termination removed. You can use the BN21W-0B Y cable with an HSZ10, HSZ40, or HSZ50 controller. Use a trilink connector with a KZTSA instead of the Y cable.
Figure 3-6 shows an HD68 trilink connector (H885), which you may attach to a KZTSA or KZPSA adapter that has its onboard termination removed, an HSZ10, HSZ40, or HSZ50 controller, or the unterminated differential side of a SCSI signal converter.
Note
If you connect a trilink connector to a SCSI bus adapter, you may block access to an adjacent PCI slot. If this occurs, use a Y cable instead of the trilink connector. This is the case with the KZPSA SCSI adapters on some AlphaServer systems.
Figure 3-7 shows a VHDCI trilink connector (UltraSCSI), which you may attach to an HSZ70 or UltraSCSI BA356 personality module that has the external SCSI bus termination disabled.
Figure 3-8 shows an Available Server shared SCSI bus that includes two DEC 3000 Model 500 systems with PMAZC TURBOchannel SCSI controllers installed, a BA350 storage shelf, and one single-ended bus. A BN21V-0B Y cable is attached to the unterminated A port in each PMAZC controller. (The unused B port is terminated with terminator part number 12-41667-01.) The BA350 storage shelf is in the middle of the bus, so its internal termination is removed. The shared bus is terminated by H8574-A or H8860-AA terminators attached to each BN21V-0B cable.
If this configuration is used and a Y cable is disconnected from a PMAZC controller, that system is not available. However, the shared SCSI bus is still operable because the bus termination is maintained, as shown in Figure 3-9.
If you have the same hardware as described in Figure 3-9, you can create an alternative hardware configuration by placing one of the systems in the middle of the shared bus and the storage shelf at the end of the bus. In this case, the BA350 storage shelf internal termination is used to terminate the bus, as shown in Figure 3-10.
However, if you have the configuration shown in Figure 3-10 and you disconnect the cable from the BA350 storage shelf, the single-ended shared bus is not terminated and ASE operation is impaired. To be able to isolate a single-ended storage shelf from a shared SCSI bus, you must connect the shelf to the single-ended side of a SCSI signal converter and attach a trilink connector to the differential side. Using this configuration, you can disconnect both the storage shelf and the signal converter from the shared bus without affecting ASE operation.
Figure 3-11 shows a hardware configuration that uses signal converters. The entire shared bus consists of three single-ended buses and one differential bus.
In Figure 3-11, there is a single-ended bus between each single-ended device (KZMSA adapter or BA350 storage shelf) and the single-ended side of a DWZZA-AA signal converter. The single-ended bus is terminated by the internal termination in the device and in the signal converter, as shown in Figure 3-12. If a cable is disconnected from the single-ended device, the single-ended bus is not terminated; however, the differential bus is not affected if you have turned off the SCSI signal converter.
In Figure 3-11, the differential bus is connected to the trilink connectors attached to the unterminated differential side of each DWZZA-AA signal converter. The differential bus is terminated by the terminators attached to the trilink connectors at the ends of the bus, as shown in Figure 3-13. If a trilink connector is disconnected from a DWZZA-AA, the differential bus is still terminated.
There are a number of requirements for connecting devices on a SCSI bus. The requirements are:
Use SCSI signal converters to connect devices with different transmission modes or data paths.
Use trilink connectors or Y cables to connect devices to the shared bus so that they can be disconnected without affecting bus termination.
Terminate each bus segment only at the ends.
Adhere to the restrictions on bus length.
All the member systems and at least one storage shelf are connected to each shared SCSI bus. You can directly connect two devices only if they have the same transmission method (single-ended or differential) and data path (narrow or wide). The SCSI cable used to connect devices depends on the transmission mode and data path of the devices.
The following sections describe how to connect devices on a shared SCSI bus. After you understand the SCSI bus connections that you need for your configuration, you can prepare the systems and storage shelves.
Single-ended devices usually have a narrow data path. However, the BA356 storage shelf is an example of a single-ended device that has a wide data path. You cannot directly connect devices with different data paths; instead, you must use two SCSI signal converters to connect the devices. See Section 3.8.3 for information about connecting narrow and wide devices.
Figure 3-14 shows the cable required for a bus connection between two single-ended, narrow devices, which include:
PMAZC TURBOchannel SCSI controller and KZMSA XMI to SCSI adapter
BA350 and BA353 storage shelf
Single-ended side of a DWZZA signal converter
Figure 3-15 shows the cable required for a bus connection between two single-ended, wide devices, which include:
BA356 storage shelf
Single-ended side of a DWZZB signal converter
Differential devices always have a wide data path. Figure 3-16 shows the cable required for a bus connection between two differential, wide devices, which include:
KZTSA and KZPSA adapters
HSZ10 and HSZ40 controllers
Differential side of a SCSI signal converter
Use a SCSI signal converter to connect devices with different transmission modes or data paths. See Section 3.6 for detailed information about using SCSI signal converters.
Figure 3-17 shows how to connect a device with a single-ended and narrow SCSI interface (for example, a BA350 or BA353 storage shelf) to a device with a differential and wide SCSI interface (for example, a KZPSA adapter) by using a DWZZA-VA signal converter installed in a disk slot.
If instead of using a DWZZA-VA you use a standalone DWZZA-AA signal converter, connect the DWZZA-AA to the single-ended, narrow device using a BN21R or BN23G cable.
Figure 3-18 shows how to connect a device with a single-ended and wide SCSI interface (for example, a BA356 storage shelf) to a device with a differential and wide SCSI interface by using a DWZZB-VW signal converter, which is installed in a disk slot.
If instead of using a DWZZB-VW you use a standalone DWZZB-AA signal converter, connect the DWZZB-AA to the single-ended, wide device using a BN21K or BN21L cable.
To connect two devices with single-ended SCSI interfaces, connect each device to a SCSI signal converter, and then connect the differential sides of the signal converters by using a BN21K or BN21L cable.